Skip to content

feat(vllm): auto-prepare trusted dual DGX Stations - #7030

Merged
cv merged 117 commits into
mainfrom
feat/dual-dgx-station-inference
Jul 28, 2026
Merged

feat(vllm): auto-prepare trusted dual DGX Stations#7030
cv merged 117 commits into
mainfrom
feat/dual-dgx-station-inference

Conversation

@ericksoa

@ericksoa ericksoa commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Station Express detects one pretrusted DGX Station GB300 peer across two direct private /30 rails and prepares the qualified pair for distributed Nemotron 3 Ultra inference. Without a qualified pair, Express keeps the existing single-Station Ultra path. The dual-Station path remains Deferred and does not change the single-Station support status. Operators still own rail configuration, network isolation, SSH trust, firewall policy, and reboots.

Changes

  • Derive only the mathematical peer on each of two active private /30, 400 Gb/s, MTU-9000 CX-8 rails. The installer does not scan subnets or enroll SSH trust.
  • Require a pretrusted reciprocal peer and validate its rail addresses, MACs, routes, neighbors, jumbo reachability, Station identity, and distinct GB300 GPU UUID before selecting distributed inference.
  • Keep NEMOCLAW_DGX_STATION_PEER, NEMOCLAW_VLLM_MODEL, and --station-deepseek authoritative and fail closed on conflicting intent.
  • Reuse scripts/prepare-dgx-station-host.sh on both hosts with exact-byte hashing and strict noninteractive SSH execution.
  • Preserve owner-only resume state across operator-managed reboots, bound to the revision, helper digest, SSH host-key identity, GPU UUIDs, and reciprocal rail identity.
  • Run the distributed Ultra path with the pinned vLLM image, Ray 2.56.0, PP=2/TP=1, FP8 KV cache, prefix caching, and served alias nemotron-ultra. Single-Station Ultra continues to serve nvidia/nemotron-3-ultra-550b-a55b.
  • Bind distributed serving to the direct trusted rails, retain bearer authentication and lifecycle ownership, and fail closed on unhealthy or ambiguously owned local-provider state.
  • Recover a validated running managed pair without restarting it, persist owner-only cleanup ownership, preserve that ownership for gateway-scoped uninstall, and remove the exact worker and head containers first during full uninstall.
  • Harden managed recovery so empty or non-200 health probes fail, API-key creation requires O_NOFOLLOW, and only explicit managed bindings enter managed lifecycle handling.
  • Complete fresh peer preparation by permitting noninteractive --check to use sudo only for read-only Docker inventory, reopening SSH after peer login-required status, and binding both controller UIDs after pair qualification.
  • Normalize every guarded installer and Local vLLM continuation state path when HOME has a trailing slash and accept physical iproute2 route JSON that omits redundant filtered fields while retaining exact direct-route, device, scope, MAC, neighbor-state, and jumbo-frame checks.
  • Add focused installer, peer-selection, model-staging, SSH-binding, authentication, lifecycle, rollback, ownership, and documentation-contract coverage.
  • Update Station preparation, vLLM, uninstall, command-reference, host-state, and pinned prompt documentation for topology selection and its security boundary.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Aaron accepted the documented trusted two-host boundary. The completed physical qualification answers the earlier evidence blocker; maintainer review refresh remains pending.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: pass
  • Evidence: Exact-head review found no blocking documentation or writing issue. The refreshed head is a mechanical merge from main; its readiness and E2E-support changes do not alter dual-Station behavior, and the seven reviewed dual-Station documentation files are byte-identical to the prior reviewed head. Installer integration passed 521 tests with 2 intentional skips, focused CLI passed 103/103, E2E support passed 19/19, the integration workflow test passed 3/3, CLI typecheck passed, and the diff check passed.
  • Agent: Codex Desktop Documentation Writer Review

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit: final behavior head 4615f2523d179da6d872fe33b1d226804b36c58f; cold-install head dd03ed8e45fb0e5f1e1ffa7f1fdeb65d5e6a0a94. Current head f82e69259dbf124491d645abb0d73d7cf8214704 adds only documentation, current-main integration, CI fixture/module-reference changes, readiness host-observation reporting, and E2E restore-result classification after the behavior head.
  • Station profile/scenario: two physical DGX Station GB300 systems running the June 2026 NVIDIA AI Developer Tools 7.5 image, connected by two direct 400 Gb/s MTU-9000 /30 rails. Product-default OpenClaw + Nemotron 3 Ultra completed reciprocal preparation, authenticated two-node Ray/vLLM serving, chat/tool smoke, managed-runtime restart recovery, warm reuse, durable cleanup ownership, and exact-pair rollback. The worker's auxiliary RTX PRO remained excluded from managed inference. All observed volatile ECC counters ended at corrected/uncorrected 0/0.
  • Evidence: complete physical lifecycle receipt and timings; accepted trusted-boundary decision; preserved fabric qualification.
  • Result: Pass. Cold install reached ready in 24m06.994s; chat completed in 5.57s; tool smoke completed in 23.50s; managed-runtime restart recovery completed in 9m50s; exact warm reuse completed in 2m51.300s; the post-reuse prompt completed in 7.57s; full rollback completed in 10.900s. Rollback removed the exact pair and receipt while preserving the image, model caches, Docker, Toolkit, OpenShell, controller binding, driver, and rail configuration. No credential was persisted or included in evidence.

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — justification: current merged-base installer integration passed 521 tests with 2 intentional skips; focused dual-runtime CLI tests passed 83/83; CLI typecheck, Biome, shell syntax, diff checks, and hooks passed.
  • Applicable broad gate passed — the complete installer-integration project and focused merged-base checks passed; GitHub aggregate checks are running on exact head f82e69259.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only) — build passed with 0 errors and 2 existing Fern warnings.
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Aaron Erickson aerickson@nvidia.com
Signed-off-by: Senthil Ravichandran senthilr@nvidia.com

ericksoa and others added 15 commits July 15, 2026 20:50
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Senthil Kumar Ravichandran <senthilr@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds DGX Station dual-peer qualification, secure SSH and resume state, managed two-node vLLM lifecycle orchestration, onboarding integration, updated documentation, and a guarded fixture-backed simulator with extensive tests.

Changes

Dual-Station installer and preparation

Layer / File(s) Summary
Peer qualification and resume flow
scripts/install.sh, scripts/lib/dgx-station-peer.mts, scripts/prepare-dual-dgx-station.mts, scripts/prepare-dgx-station-host.sh
Discovers reciprocal trusted Station peers, validates rails and connectivity, persists revision-bound resume state, supports reboot-required status 10, and runs host preparation in explicit modes.
Managed vLLM runtime
src/lib/inference/vllm-station-cluster.ts, src/lib/inference/vllm-station-cluster-lifecycle.ts, src/lib/inference/vllm-station-model-staging.ts, src/lib/inference/vllm.ts
Builds validated dual-node plans, stages pinned model snapshots, performs GPU/runtime preflight, starts worker/head containers, verifies readiness and authentication, and rolls back exact owned resources.
SSH binding and Docker transport
src/lib/inference/vllm-station-ssh-binding.ts, src/lib/inference/vllm-docker-env.ts, src/lib/inference/vllm-api-key.ts
Persists immutable host-key bindings, validates filesystem metadata, creates sanitized local/remote Docker environments, and manages a protected shared API key.
Inference and onboarding integration
src/lib/inference/local.ts, src/lib/inference/context-window.ts, src/lib/onboard/*, src/lib/inference/vllm-models.ts
Routes local vLLM probing through managed endpoints when available, passes explicit managed credentials to validation, and adds the pinned distributed Nemotron command and gated-model handling.
Simulation and tests
scripts/simulate-dual-station.mts, src/lib/inference/*simulator*, test/install-station-pair-preparation.test.ts, vitest.config.ts
Adds a restricted simulator command, in-memory dual-Station lifecycle fixtures, installer project routing, and regression coverage for qualification, lifecycle, authentication, and resume behavior.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#5927 — Shares the inference-selection validation boundary and its dependency-injection changes.
  • NVIDIA/NemoClaw#6883 — Overlaps the DGX Station express-install and Nemotron model-selection logic.
  • NVIDIA/NemoClaw#6991 — Extends the same Station host-preparation and controller-binding workflow.

Suggested labels: area: ci, area: e2e, area: install

Suggested reviewers: cv, sandl99, hoyalim

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 10.36% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately captures the main change: automatic preparation for trusted dual DGX Station setups.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/dual-dgx-station-inference

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit f82e692 in the feat/dual-dgx-statio... branch remains at 96%, unchanged from commit 2540869 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit f82e692 in the feat/dual-dgx-statio... branch is 80%. The coverage in commit 6c43fe2 in the main branch is 81%.

Show a code coverage summary of the most impacted files.
File main 6c43fe2 feat/dual-dgx-statio... f82e692 +/-
src/lib/inference/vllm.ts 90% 88% -2%
src/lib/inferen...odel-staging.ts 0% 65% +65%
src/lib/inferen...er-lifecycle.ts 0% 76% +76%
src/lib/inferen...fecycle-lock.ts 0% 78% +78%
src/lib/inferen...vllm-api-key.ts 0% 80% +80%
src/lib/inferen...tion-cluster.ts 0% 83% +83%
src/lib/inferen...-ssh-binding.ts 0% 87% +87%
src/lib/inferen...time-receipt.ts 0% 89% +89%
src/lib/readiness/host.ts 0% 100% +100%
src/lib/inferen...test-support.ts 0% 100% +100%

Updated July 28, 2026 02:30 UTC

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-inference, cloud-onboard, security-posture, inference-routing, network-policy, onboard-repair, onboard-resume

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Comment thread src/lib/inference/vllm-api-key.ts Fixed
Comment thread src/lib/inference/vllm-station-cluster-lifecycle.ts Fixed
Comment thread src/lib/inference/vllm-station-cluster-lifecycle.ts Fixed
@ericksoa ericksoa added the feature PR adds or expands user-visible functionality label Jul 16, 2026
@ericksoa ericksoa self-assigned this Jul 16, 2026
@ericksoa ericksoa added platform: dgx-station Affects DGX Station hardware or workflows provider: vllm vLLM local or hosted provider behavior area: inference Inference routing, serving, model selection, or outputs area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow labels Jul 16, 2026
senthilr-nv and others added 4 commits July 16, 2026 11:22
Signed-off-by: Senthil Kumar Ravichandran <senthilr@nvidia.com>
…st-prereqs

Signed-off-by: Senthil Kumar Ravichandran <senthilr@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
(cherry picked from commit 586a60d)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Copilot AI review requested due to automatic review settings July 27, 2026 20:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Copilot AI review requested due to automatic review settings July 27, 2026 21:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Copilot AI review requested due to automatic review settings July 27, 2026 23:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

@senthilr-nv

Copy link
Copy Markdown
Collaborator

@prekshivyas The physical and trusted-boundary blockers from the review of 39a2a0070 have now been addressed.

Physical dual-Station result

Two physical DGX Station GB300 systems on the accepted direct private /30 topology completed the product lifecycle:

  • cold Station Express install with OpenClaw and nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4;
  • exact model revision 183968f87ae4cedce3039313cac1fd43d112c578;
  • pinned image vllm/vllm-openai@sha256:2cc49b81319f7a66a33dd8bd63a7bfddae079122b33ce51989b6828a1f038c37;
  • reciprocal pair qualification, Ray startup, pipeline-parallel vLLM readiness, and bearer-authenticated inference;
  • real OpenClaw chat (STATION_CHAT_OK) and independently evaluated tool artifact (STATION_TOOL_OK);
  • ordered managed-runtime restart recovery with the same container IDs, image, cluster identity, model route, and ECC state;
  • exact running-pair warm reuse with durable owner-only cleanup ownership;
  • full rollback that removed the exact worker and head containers first while preserving model caches, image, Docker, Toolkit, OpenShell, controller binding, driver, and rail configuration.

Timings:

  • cold install to ready: 24m06.994s (installer UI: 23m48s);
  • chat: 5.57s;
  • tool: 23.50s;
  • managed-runtime restart recovery: 9m50s;
  • final exact warm reuse: 2m51.300s (installer UI: 159s);
  • post-reuse real prompt: 7.57s;
  • full rollback: 10.900s.

All observed GB300 and auxiliary RTX PRO volatile ECC counters ended at corrected/uncorrected 0/0. No credential was persisted or included in evidence.

The cold product run used dd03ed8e45fb0e5f1e1ffa7f1fdeb65d5e6a0a94. Physical warm reuse and rollback were repeated on final behavior head 4615f2523d179da6d872fe33b1d226804b36c58f, which added the fail-closed persisted cleanup receipt. The current PR head 8011ae2b37e1aa5b452e613ad06980a583dbe54f contains only documentation clarification, current-main integration, a launch-schema fixture correction, and a net-neutral module-reference consolidation after that behavior head. Full installer integration passes 521 tests with 2 intentional skips; focused dual-runtime CLI tests pass 83/83; CLI typecheck and hooks pass. Both new commits are GitHub Verified.

The implementation and documentation remain within the accepted Deferred boundary: exactly two trusted Stations, direct private /30 rails, operator-owned isolation/firewall/SSH/reboots, no shared-network discovery, and no support-status promotion. This AI Developer Tools path did not require a physical host reboot; the evidence claims managed-runtime restart recovery, not host-reboot recovery.

Please re-review or dismiss the stale changes-requested state after the current CI run settles.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

@cjagwani cjagwani mentioned this pull request Jul 28, 2026
23 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: inference Inference routing, serving, model selection, or outputs area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow feature PR adds or expands user-visible functionality platform: dgx-station Affects DGX Station hardware or workflows provider: vllm vLLM local or hosted provider behavior v0.0.97 Release target v0.0.98 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants